Method and system for geometric referencing of multi-spectral data

ABSTRACT

A sensing device for obtaining geometric referenced multi-spectral image data of a RoI in relative movement with respect to the device, the sensing device having: a first and second 2D sensor element, the device obtaining subsequent multi-spectral images during the relative motion of the Rol thus providing distinct spectral information for parts of a Rol using the first element; and providing, using the second element, an image of the Rol for generating geometric referencing information to be coupled to the distinct spectral information; each element capturing a sequence of frames at a respective frame rate. The first frame rate is higher than the second frame rate. The device generates intermediate geometric referencing information to be coupled to frames of the first sequence of frames for which no synchronous frame from the second sequence of frames is available, derived from temporally similar frames from the second sequence of frames.

FIELD OF THE INVENTION

The invention relates to the field of image capturing e.g. in aerial orindustrial imaging. More particularly, the present invention relates tosensing systems for obtaining multi-spectral images, correspondingimaging systems and methods for using them.

BACKGROUND OF THE INVENTION

Hyperspectral imaging is a form of spectral imaging wherein informationfrom across the electromagnetic spectrum is collected in many narrowspectral bands and processed. From the different spectral images thatare collected, information of the objects that are imaged can bederived. For example, as certain objects leave unique spectralsignatures in images which may even depend on the status of the object,information obtained by multi-spectral imaging can provide informationregarding the presence and/or status of objects in a region that isimaged. After selection of a spectral range that will be imaged, asspectral images in this complete spectral range can be acquired, onedoes not need to have detailed prior knowledge of the objects, andpost-processing may allow to obtain all available information. Whereasoriginally hyperspectral remote sensing was mainly used for mining andgeology, other applications such as ecology, agriculture andsurveillance also make use of the imaging technique.

Some agricultural and ecological applications are known whereinhyperspectral remote sensing is used, e.g. for monitoring thedevelopment and health of crops, grape variety detection, monitoringindividual forest canopies, detection of the chemical composition ofplants as well as early detection of disease outbreaks, monitoring ofimpact of pollution and other environmental factors, etc. are some ofthe agricultural applications of interest. Hyperspectral imaging also isused for studies of inland and coastal waters for detecting biophysicalproperties. In mineralogy, detection of valuable minerals such as goldor diamonds can be performed using hyperspectral sensing, but alsodetection of oil and gas leakage from pipelines and natural wells areenvisaged. Detection of soil composition on earth or even at otherplanets, asteroids or comets also are possible applications ofhyperspectral imaging. In surveillance, hyperspectral imaging can forexample be performed for detection of living creatures.

In some applications, multi-spectral data can be obtained by collectinga full two dimensional image of a region in one spectral range ofinterest and by subsequently collecting other full two dimensionalimages of that region in other spectral ranges of interest wherebyspectral filters are switched in between. This way of data collectionnevertheless is not always possible, especially when the region ofinterest and the imaging system undergo a large relative movement withrespect to each other.

In view of the relative movement, accurate determination of positionalinformation is important for a correct interpretation of the collecteddifferent spectral data. Known systems make use of a global positioningsystem (GPS) and/or an inertial measurement unit (IMU).

International patent application publication WO 2011/073430 A1, in thename of the present applicant, discloses a sensing device for obtaininggeometric referenced multi-spectral image data of a region of interestin relative movement with respect to the sensing device. The sensingdevice comprises a first two dimensional sensor element. The sensingdevice is adapted for obtaining subsequent multi-spectral images duringsaid relative motion of the region of interest with respect to thesensing device thus providing spectrally distinct information fordifferent parts of a region of interest using different parts of thefirst sensor. The sensing device also comprises a second two dimensionalsensor element and is adapted for providing, using the second sensorelement, an image of the region of interest for generating geometricreferencing information to be coupled to the distinct spectralinformation.

The known sensor device acquires spectral data (with the first sensorelement) and geometric data (with the second sensor element) at the sameframe rate, e.g. 50 frames per second.

When the frame rate is further increased, the known sensor devicegenerates a large amount of data which can be difficult to handle andthe registration of the spectral data with the geometric data becomescomputationally more demanding.

This disadvantage can render the known sensor device less suitable forapplications which require a large number of spectral channels. Then avery high frame rate is required to ensure full spatial coverage in allthe relevant bands of the spectrum.

SUMMARY OF THE INVENTION

It is an object of embodiments of the present invention to at leastpartially overcome disadvantages of the sensor device of the prior art.

More particularly, it is an object of embodiments of the presentinvention to provide a sensor device that can be efficiently used atincreased frame rates of the spectral sensing element.

According to an aspect of the present invention, there is provided asensing device for obtaining geometric referenced multi-spectral imagedata of a region of interest in relative movement with respect to thesensing device, the sensing device comprising: at least a first twodimensional sensor element, the sensing device being adapted forobtaining subsequent multi-spectral images during said relative motionof the region of interest with respect to the sensing device thusproviding distinct spectral information for different parts of a regionof interest using the first sensor element; a second two dimensionalsensor element, the sensing device being adapted for providing, usingthe second sensor element, an image of the region of interest forgenerating geometric referencing information to be coupled to thedistinct spectral information; the first two dimensional sensor elementbeing operable to capture a first sequence of frames at a first framerate and the second two dimensional sensor element being operable tocapture a second sequence of frames at a second frame rate; wherein thefirst frame rate is higher than the second frame rate; and wherein thesensing device further comprises a processor configured to generateintermediate geometric referencing information to be coupled to framesof said first sequence of frames for which no synchronous frame fromsaid second sequence of frames is available, said intermediate geometricreferencing information being derived from one or more temporallysimilar frames from said second sequence of frames.

It is an advantage of the present invention that the second image sensorcan be operated at a reduced frame rate, resulting in a less massiveamount of data being generated. This architecture thus allows anincrease in the frame rate of the spectral sensor, which in turn allowsthe use of a spectral sensor with more wavelength channels. In this way,a very efficient hyperspectral sensing device can be provided.

In an embodiment of the sensing device according to the presentinvention, said second frame rate is selected to ensure a predeterminedamount of overlap between respective regions imaged by consecutiveframes of said second sequence of frames.

It is an advantage of this embodiment, that the second frame rate can beadjusted to provide the necessary amount of overlap to allow propergeometric referencing, while the first frame rate can be set to thevalue required to provide full spatial coverage of all the relevantwavelength channels. The first frame rate may be such that an overlap ofthe image, e.g. with at least 10%, more advantageously with at least25%, still more advantageously with at least 50% such as e.g. with 60%overlap with the previous image is established, such that informationregarding the relative change in orientation of the instrument betweensubsequent images can be detected.

In an embodiment of the sensing device according to the presentinvention, a spectral filter and the first sensor element are arrangedfor obtaining spectral information at a first wavelength or wavelengthrange using a part of the first sensor element and for obtainingspectral information at a second wavelength or wavelength range usinganother part of the first sensor element.

This is a particularly advantageous manner of implementing the firstsensor element in the system according to the present invention.

In an embodiment of the sensing device according to the presentinvention, the first sensor element and second sensor element areintegrated on the same substrate.

It is an advantage of this embodiment that the spatial relationshipbetween the first sensor element and the second sensor element is fixed,which facilitates the geo-referencing and removes the need for frequentrecalibration of the sensing device. It is a further advantage of thisembodiment that integration and fabrication on the same chip may resultin similar thermal behavior of the at least two sensors, which may be ofsignificant importance as for light weight UAVs, typically nocompensation for thermal loads on the systems can be provided in view ofweight. A similar thermal behavior of the sensors also may beadvantageous in other applications, as no or little thermal loadcompensation is required.

According to an aspect of the present invention, there is provided animaging system comprising the sensing device described above.

According to an aspect of the present invention, there is provided anaerial vehicle comprising the imaging system described above.

According to an aspect of the present invention, there is provided amethod for obtaining geometric referenced multi-spectral image data of aregion of interest in relative movement with respect to a sensingdevice, the sensing device comprising:—obtaining subsequentmulti-spectral images during said relative motion of the region ofinterest with respect to the sensing device thus providing distinctspectral information for different parts of a region of interest usingthe first sensor element; providing, using the second sensor element, animage of the region of interest for generating geometric referencinginformation to be coupled to the distinct spectral information; thefirst two dimensional sensor element capturing a first sequence offrames at a first frame rate and the second two dimensional sensorelement capturing a second sequence of frames at a second frame rate;wherein the first frame rate is higher than the second frame rate; andwherein the method further comprises generating intermediate geometricreferencing information to be coupled to frames of said first sequenceof frames for which no synchronous frame from said second sequence offrames is available, said intermediate geometric referencing informationbeing derived from one or more temporally adjacent frames from saidsecond sequence of frames.

According to an aspect of the present invention, there is provided acomputer program product comprising code means configured to cause aprocessor to carry out the method described above.

The technical effects and advantages of embodiments of the imagingsystem, the aerial vehicle, the method, and the computer program productaccording to the invention correspond, mutatis mutandis, to those of thecorresponding embodiments of the sensing device according to the presentinvention.

BRIEF DESCRIPTION OF THE FIGURES

These and other technical aspects and advantages of embodiments of thepresent invention will now be described in more detail with reference tothe accompanying drawings, in which:

FIG. 1 shows a schematic overview of a sensing device for obtaininggeo-referenced multi-spectral data as may be used in an embodiment ofthe present invention;

FIG. 2 shows a schematic illustration of the lay-out of sensor elementson the sensing device for obtaining geo-referenced multi-spectral data,as disclosed in WO 2011/073430 A1, which can be improved according tothe present invention;

FIG. 3 illustrates a number of hyperspectral images as can be used in asystem according to an embodiment of the present invention;

FIG. 4 shows an imaging system comprising a sensing device for obtaininggeo-reference multi-spectral image data according to an embodiment ofthe present invention;

FIG. 5 represents a timing diagram of the capturing of spectral (S) andgeometric (G) frames according to an embodiment of the presentinvention;

FIG. 6 represents a flow chart of an embodiment of the presentinvention;

FIG. 7 represents a flow chart of an algorithm for use in an embodimentof the present invention;

FIG. 8 provides a first graph illustrating filtering and interpolationalgorithms used in embodiments of the present invention; and

FIG. 9 provides a second graph illustrating filtering and interpolationalgorithms used in embodiments of the present invention.

In the different drawings, where applicable, the same reference signsrefer to the same or analogous elements.

DETAILED DESCRIPTION OF THE INVENTION

For illustrative purposes, the invention will be described withreference to the sensing device of the aforementioned internationalpatent application publication WO 2011/073430 A1, but this is donewithout loss of generality. The characterizing features of the presentinvention can be combined with features of various embodiments of theknown sensing device, as described in more detail in the followingdescription. Details of the known sensing device are omitted where thisis not necessary for the understanding of the present invention; thedescription of WO 2011/073430 A1 is hereby incorporated by reference forthe purpose of guiding the implementation of corresponding embodimentsof the present invention.

Where in the present invention reference is made to a two dimensionalmulti-spectral image, reference is made to an m×n pixelated imagecomprising information regarding one part of a region of interest imagedat one wavelength or spectral region and comprising informationregarding at least another part of a region of interest imaged at adifferent wavelength or spectral region. Whereas the obtained spectralinformation within one spectral region may be a line, group orsub-matrix of pixels, the overall underlying pixelated sensor typicallyis a two dimensional spectral sensor. Embodiments according to thepresent invention may be applicable in a broad spectral range ofelectromagnetic radiation. Particular ranges that may be covered arevisual and near IR (VNIR), typically considered to be in the range 400nm to 1000 nm), short wave infrared, thermal infrared, etc., embodimentsof the present invention not being limited to the exemplary rangesgiven. Where in embodiments of the present invention reference is madeto a multi-spectral image or multi-spectral image data, reference ismade to data comprising separate information regarding a region ofinterest for at least two different wavelengths or wavelength regions.Hyperspectral images or image data refer to data comprising separateinformation for a large number of wavelength or wavelength regions.Unless otherwise noted, references to multi-spectral images includehyperspectral images.

Where in embodiments according to the present invention reference ismade to geo-referencing or geometric referencing of a point or object inthe region of interest, reference is made to the existence of the pointor object in a region of interest in physical space. It refers toestablishing the location in terms of map projections or coordinatesystems. The latter may for example include positional information, e.g.relative positional information. Such positional information may be(x,y) related positional information, but also z-related positionalinformation such as height or relative height. It is not only applicableto aerial photography, aerial imaging or satellite imaging, where it isoften referred to as geo referencing, but also in other applications,such as for example in industrial inspection.

In a first aspect, the present invention relates to a sensing device forobtaining geometric referenced multi-spectral image data. The sensingdevice may especially be suitable for hyperspectral imaging, althoughembodiments of the present invention are not limited thereto. Thesensing device according to embodiments of the present invention areespecially suitable for obtaining geometric referenced multi-spectralimage data, using a sensing device and a region of interest in relativemovement with respect to each other, which is for example the case whenimaging from air is performed or when imaging using a top view isperformed. The sensing device according to embodiments of the presentinvention comprises a single substrate, e.g. a single chip. Thesubstrate may be any type of substrate, such as for example a glasssubstrate, a polymer substrate, a semiconductor substrate, etc. In someadvantageous embodiments, the substrate may be a semiconductor chip,providing the possibility of using semiconductor processing steps forintegration of the sensor elements. The single chip comprises at least afirst two dimensional sensor element, whereby the sensing device isadapted for providing spectrally different information for differentparts of a region of interest using the first two dimensional sensorelement. The single chip also comprises a second two dimensional sensorelement, whereby the sensing device is adapted for providing geometricreferencing information of the region of interest using the secondsensor element. The geometric referencing information advantageously maybe coupled to the spectral information obtained by the sensing device.It is an advantage of embodiments according to the present inventionthat at least one first and second sensor element are processed on thesame chip. The latter allows for accurate alignment of the sensorelements, such that little or no subsequent alignment for positioningthe sensor elements with respect to each other is required.

By way of illustration, the present invention not being limited thereto,an exemplary sensing device as well as an imaging system comprising sucha sensing device will be further discussed with reference to FIG. 1 toFIG. 3, embodiments of the present invention not being limited thereto.

In FIG. 1, a sensing device 100 according to an embodiment of thepresent invention is shown, wherein the sensing device 100 comprises atleast one first sensor element 112 and a second sensor element 122processed on the same chip, i.e. processed on the same substrate 102.The first sensor element 112 and second sensor element 122 and optionalfurther sensor elements thus may be homogeneously or heterogeneouslyprocessed sensor elements, processed on the same substrate 102.Advantageously, the sensor elements are homogeneously processed sensorelements 112, 122 on the same substrate 102. The sensor elements 112,122 may be integrated on the same substrate 102 whereby the differentlayers constituting the different sensor elements are processed for bothsensor elements 112, 122 using the same processing technology, forexample—but not limited to—CMOS processing technology. The sensorelements typically may comprise a plurality of pixels. The pixelstypically may be arranged in a matrix form in a number of columns androws, although the invention is not limited thereto. The sensor elementsmay be referred to as frame sensor elements, as the sensor elements aretwo dimensional sensor elements, comprising e.g. a matrix of sensorpixels m×n. The two sensor elements may be selected so that at least oneof the number of pixels in a row or the number of pixels in a column isthe same for both sensors. In an advantageous embodiment, the sensorelements may comprise a high number of pixels in one direction forimaging simultaneously a relatively wide region of interest with arelatively high spatial resolution. If for example the sensing device isused for detecting or monitoring a region of interest using an UAV, apreferred scanning width may be at least 1000 m, more advantageously atleast 2000 m, still more advantageously at least 3000 m. Taking intoaccount a preferred ground resolution of at least 1 m, moreadvantageously at least 50 cm, still more advantageously at least 30 cm,the number of pixels in one direction may in some examples be at least1000, in other examples at least 4000, in still other examples 10000. Byway of illustration, embodiments of the present invention not beinglimited thereby, an example of a lay-out for the sensor elements 112,122 on the substrate is shown in FIG. 2. The sensor elements 112, 122advantageously are surface aligned. The distance between the two sensorsmay be smaller than 1 mm, although embodiments of the present inventionare not limited thereby.

The sensing device 100 furthermore comprises drive and read-outcircuitry for driving the sensor elements 112, 122. The drive andread-out circuitry 130 may be adapted for driving the sensor elements112, 122 differently from each other. The drive and read-out circuitry130 may be a drive and read-out circuit as known from prior art, wherebythe drive and read-out circuitry 130 may comprise components such asamplifiers, switches, a buss, etc. In some embodiments, the pixeldesign, the column structure and the bus driver are laid out so that amultiplexer following the bus can be avoided, resulting in a betterimage quality. The drive and read-out circuitry also may be adapted forreading out the sensor elements 112,122. The read-out may be optimizedfor efficient and fast reading out. For example in a 10000×1200 sensorthe frame rate at full resolution may be at least 35 frames per second,e.g. at least 50 frames per second. The driving and reading out also maybe performed by different components, i.e. a separate drive circuitryand separate reading-out circuitry may be provided. The sensors may beequipped with shutters so that fast shutting, e.g. electronic shutting,can be obtained. The sensor elements as well as the driving and read-outcircuitry may be processed on the same chip or die using semiconductorprocessing, such as for example CMOS technology, embodiments of theinvention not being limited thereto.

According to embodiments of the present invention, the sensing device isadapted for providing different spectral information for different partsof a region of interest using the first two dimensional sensor element.The sensing device may thus be adapted for generating a multi-spectralimage. In some embodiments, the sensing device may be adapted forgenerating hyperspectral data, i.e. in many narrow spectral bands. Asthe first sensor element according to embodiments of the presentinvention is a two-dimensional sensor element and as different spectralinformation is to be captured, typically part of the sensor element maybe used for obtaining spectral information at a first wavelength or in afirst wavelength region for one part of the region of interest, and atleast one other part of the sensor element may be used for obtainingspectral information at least a second wavelength or in at least asecond wavelength region for at least another part of the region ofinterest. In some embodiments, different lines of the sensor element maybe used for gathering data at different spectral wavelengths or indifferent spectral wavelength regions. In other embodiments for exampledifferent blocks of the sensor element may be used for sensing differentspectral data or different columns may be used for sensing differentspectral data. In order to capture different spectral information, amulti-spectral filter 114, advantageously hyper spectral filter, may bepresent. The multi-spectral filter 114 forms together with the firstsensor element 112 and the drive and read-out circuitry or part thereoffor controlling the first sensor element 112, the first sensor. Themulti-spectral filter may be directly applied to the first sensorelement, e.g. mechanically behaving as a single element. Alternatively,the two components may be separate from each other, but configured orarranged so that appropriate filtering is obtained. By way ofillustration, some examples of multi-spectral sensors are now discussed.

By way of illustration a set of hyperspectral data that can be obtainedusing a sensor according to an exemplary embodiment of the presentinvention is shown in FIG. 3. Whereas reference is made to hyperspectraldata, the latter also applies to multispectral data. FIG. 3 illustratesa plurality of subsequent hyperspectral images recorded in m subsequenttime spans, whereby the spectra are recorded for a relative movementbetween region of interest and sensing or imaging system correspondingwith a total shift over a distance xm-x1 travelled during the total ofthe subsequent time spans.

FIG. 3 illustrates m hyperspectral images, each image consisting of mlines, wherein line L_(j) comprises information of wavelength λ_(j) ore.g. of spectral band λ_(j)-λ_(j−1). The different images are recordedwithin m subsequent time frames. By way of illustration, the imaging ofa physical position at coordinates x_(p) and y_(q) of the region ofinterest is indicated throughout the different hyperspectral images. Itcan for example be seen that in the information regarding the physicalposition at coordinate x₁ for different y_(q) coordinates of the regionof interest is in the first hyperspectral image HI₁ found in line 1, inthe second hyperspectral image HI₂ found in line 2, in the thirdhyperspectral image HI₃ found in line 3, . . . and in the m′thhyperspectral image HI_(m) found in line m. In each of thesehyperspectral images this information is imaged for a differentwavelength or wavelength region. Seen in an alternative way, informationregarding the region of interest imaged at wavelength λ_(m) or in acorresponding spectral band λ_(m)-λ_(m−1), can be found by combining theinformation in line m of subsequent hyperspectral images as adjacentimage lines, the ensemble over the m hyperspectral images forming an m×nimage of the region of interest imaged at wavelength λ_(m) or in acorresponding spectral band λ_(m)-λ_(m−1). Similarly, the other lines ofm subsequent hyperspectral images contain information regarding a regionof interest at a different wavelength or in a different spectral band.The latter illustrates how hyperspectral images provide informationregarding different spectral wavelengths or in different spectral bandsand how subsequent hyperspectral images recorded during relativemovement of region of interest and sensing system can provide an imageof the full region of interest for different wavelengths or in differentspectral bands. It is to be noticed, that whereas the principle isillustrated for subsequent lines covering different wavelengths,embodiments of the present invention are not limited thereto, and thevariety of spectral information also may be obtained in otherdirections, e.g. varying spectral info for subsequent columns.Furthermore, whereas the principle is illustrated for a sensor whereineach line corresponds with a different wavelength or spectral region,embodiments of the present invention are not limited thereto and severallines of the spectral image may correspond with the same wavelength orspectral region. It is a characteristic of a spectral image that theimage comprises information regarding at least two different wavelengthsor spectral regions. Capturing of information using the principle asdescribed above has the advantage that using a two dimensional sensorelement, two dimensional images are recorded at different wavelengths orspectral regions, i.e. resulting in three dimensional information (twopositional dimensions, one spectral dimension). In other words, in someembodiments according to the present invention, the sensor element forspectral data may be used as a set of line or block sensing sub-elementseach sub-element recording positional information for a given wavelengthor in a spectral region, whereby recording over time during relativemovement of the region of interest with respect to the sensor elementcorresponds with scanning different positions of a region of interest.As described above, the sensing device 100 furthermore comprises asecond two-dimensional sensor element 122 that forms, together with thedriving and read-out circuitry or part thereof for driving the secondtwo-dimensional sensor element 122 the second sensor 120. The secondsensor 120 may be adapted for obtaining an image of the region ofinterest from which geo-referencing information can be obtained. Thesecond sensor 120 may be adapted for providing a high resolution image,providing detailed geometric information, e.g. geographical information,regarding the region of interest. Images obtained via the second sensor120 may allow to derive tie points in the imaged region of interest.

The frequency at which the images are captured with the second sensormay be such that an overlap of the image, e.g. with at least 10%, moreadvantageously with at least 25%, still more advantageously with atleast 50% such as e.g. with 60% overlap with the previous image isestablished, such that information regarding the relative change inorientation of the instrument between subsequent images can be detected.The obtained information regarding rotation may be used as geometricreferencing information, according to embodiments of the presentinvention, for coupling to the multi-spectral data obtained using thefirst sensor 110, so that geo-referenced multi-spectral data can beobtained.

According to the invention, the frequency at which the images arecaptured with the second sensor is lower than that at which images arecaptured with the first sensor. Preferably, the image acquisitionfrequency of the second sensor is an integer fraction of the imageacquisition frequency of the first sensor, such that images acquiredwith the first sensor periodically coincide in time with images acquiredwith the second sensor. For the images of the first sensor for which nosynchronous image of the second sensor exists, interpolated geometricdata has to be used. This interpolated geometric data is obtained fromthe second-sensor images that are closest in acquisition time to thetargeted first-sensor image, preferably the second-sensor imagesimmediately preceding and immediately following the targetedfirst-sensor image. Such a difference in image acquisition frequenciesis schematically illustrated in FIG. 5.

In embodiments of the present invention, an interpolation algorithm isused which is specifically suited for platform attitude interpolation.The parameters to be interpolated include the various angles thatdescribe the attitude of the sensing device. Accordingly, certainembodiments at least partially use interpolation of attitude data inangular coordinates.

In embodiments of the present invention, the interpolation algorithmuses first derivatives (angular and/or linear velocity) and secondderivatives (angular and/or linear acceleration) to predict the attitudeof the sensing device at moments between captured images, taking intoaccount the laws of mechanics that govern the motion of the sensingdevice and the platform on which it is mounted (e.g., an aerialvehicle). These first and second derivatives may be obtained from aninertial measurement unit (including GPS, gyroscopes, accelerometers,etc.).

The purpose of the interpolation algorithm is to obtain accurateestimates on position and external orientation of the platform for everyfirst sensor image (spectral image), on the basis of the (lessfrequently acquired) second sensor images. The inputs for this may beobtained from two independent sources:

-   -   Filtered GPS/IMU (short time intervals)    -   Using GPS (for position) and IMU (for attitude): They provide        “raw” measurement data at small time intervals. This data is        typically noisy and individual measurements are not very        accurate. It is customary to filter the data using “Kalman        filtering” (which is the optimal filter for this type of data)        to achieve smooth “most plausible” estimates at all points in        time. This filtering is shown as “static correction” in the        diagram of FIG. 7.    -   Image based (high accuracy)    -   Using image data: (high quality, high spatial resolution second        sensor images). Dedicated points in the images are matched using        features (ground control points). This can yield very accurate        estimates of position and attitude for the time points for which        there is a G-frame available.

Optimal results for all first sensor images (i.e., all time points) canbe obtained by combining the two sources of information. We list threepossible methods:

-   -   1. Correction: the filtered results are used, and every time an        image based result is available, the filtered result is        corrected to the image based value by applying a simple offset.        This offset is kept constant for the next filtered results. When        the next image based result is available the error with the        obtained result is calculated. This is fed back in the loop to        set the new offset. The process is depicted in FIG. 7.    -   An example result for a single variable is shown in the graph        presented in FIG. 8. The measured data are the dots, showing        substantial noise between subsequent points. The bottom dotted        curve represents the Kalman-filtered results. At point 1, 11 and        21, there is also an image based result. The offset is set at        point 1 and applied up to point 10. At point 11 a new offset is        calculated and applied until point 21. The results are shown in        the solid curve. At point 11 and 21, new offsets are applied,        which result in a sharp jumps in the curve.    -   2. Interpolation: Adjust the Filtered GPS/IMU result to the        values of the image based result for the available time points.        Instead of applying fixed offsets as in the previous method, the        offsets are calculated at all points where possible, and        interpolated for all points in between. A simple linear        interpolation can be assumed.    -   An example of this is shown in the graph presented in FIG. 9.        The dashed curve now shows the linear interpolation between the        offsets. In the final result the linear behavior of the filtered        result is replaced by the interpolated image based result. It is        shown in the solid line in the graph. This matches the image        based points, and follows the filtered the shape of the filtered        curve in between. The advantage is that jumps in the result are        avoided. The disadvantage is that the intermediate results can        only be calculated after the next image based result is        available.    -   3. Add image based results to Kalman filtering: The image based        results can be simply added to the set of GPS/IMU raw data. When        proper weights are given to acknowledge the higher accuracy, the        Kalman filtering will take this into account and use the        information optimally. This leads in principle to superior        results.

Embodiments of the present invention also relate to an imaging system. Aschematic representation of an imaging system 200 comprising a sensingsystem according to embodiments of the present invention is shown inFIG. 4 by way of example. The imaging system 200 comprises a sensingdevice 100 as described for example above. The imaging system 200furthermore comprises optical elements for guiding radiation to the twosensing elements of the sensing device 100. Such optical elements mayfor example comprise at least one lens 210 for capturing the radiationto be collected and focusing the radiation onto the sensor elements. Insome embodiments, a single lens 210 may be used for collecting theradiation for both sensor elements, whereas in other embodimentsdifferent lenses may be used for the different sensor elements. In someembodiments according to the present invention, the collected radiationmay be split to the two sensor elements using a radiation splitter, suchas for example a beam splitter 220. Alternatively, or in additionthereto, the configuration of the sensor elements 112, 122 processed onthe same substrate 102 may allow for taking into account positionalinformation between the sensor elements when correlating the imagesobtained using the two sensor elements.

The imaging system furthermore may comprise an image processor 230 forcorrelating the images obtained with the first sensor 110 and the secondsensor 120. The image processor may for example correlate geometricinformation, e.g. positional information, obtained with the secondsensor 120 with spectral information obtained in different spectralchannels in the first sensor 110, so that accurate hyperspectralinformation is obtained. Such image processing may be performed in asingle processor or in a plurality of processors. The processing may beperformed after the full set of images have been captured, although insome embodiments substantially direct processing may be performed, assoon as all information regarding the same region of interest iscaptured in both sensors 110, 120. A more detailed description of theimage processing that may be performed by a processor 230 according toembodiments of the present invention will further be discussed laterwith reference to FIG. 6, illustrating standard and optional steps of anexample of a method for sensing according to an embodiment of thepresent invention.

The imaging device furthermore may comprise a global positioning systemfor providing GPS data and/or an inertial measurement unit for providinginertial data regarding the imaging system. Such components may assistin providing approximate geo-referencing data, which may assist inderiving geo-referenced spectral-data based on the image obtained withthe second sensor 120.

In one aspect, the present invention thus also relates to an imagingsystem as described above comprising a sensing device as describedabove. In another aspect the present invention also relates to anindustrial system or unmanned aerial vehicle (UAV) comprising such animaging system for monitoring, imaging or inspection. It thereby is anadvantage of embodiments according to the present invention that thesensing device comprises the two sensing elements on the same sensor,such that thermal load due to temperature variation or such thatenvironmental conditions have less influence on the obtained result. Instill another aspect, the present invention relates to a method forobtaining image data regarding a region of interest. It thereby is anadvantage of embodiments according to the present invention thatmulti-spectral data of a region of interest can be obtained with highgeometric accuracy, e.g. geographic accuracy, e.g. a geometric accuracythat is significantly higher than can be obtained using globalpositioning and/or inertial measurement systems alone. The method isespecially suitable in applications where multi-spectral data of aregion of interest are obtained using sensing device that undergo arelative movement with respect to the region of interest, such as forexample in case aerial imaging is performed or e.g. during industrialinspection of moving products. In case of aerial imaging, the methodfurthermore also is especially suitable for use in unmanned aerialvehicles (UAV), as the method can be performed using components low inweight, which is a major requirement if unmanned aerial vehicles are tobe used or are to be used for a longer time. More particularly, thelower the weight to be carried, the lower the power consumption requiredand the longer flying times can be obtained with the unmanned aerialvehicles.

In order to further illustrate standard and optional features of amethod according to an embodiment of the present invention, FIG. 6illustrates a detailed flow chart of an exemplary method for obtainingimage data. The exemplary method thereby is adapted for capturing atleast one two-dimensional image of the region of interest for derivinggeometric referencing information, and for capturing hyperspectralimages using a system as described above. More particularly, in thepresent example, the different hyperspectral images are obtained duringrelative movement of the region of interest with respect to the imagingsystem. The hyperspectral images are taken at a higher rate than thegeometric reference images, which rates are preferably integer multiplesand may be derived from a common synchronization block 405. Using onesensor, image acquisition for obtaining a two dimensional image of aregion of interest is performed in step 430. Such image acquisitionincludes acquisition of a set of frame images FI₁, FI₂, . . . FI_(n),whereby n images are captured, as indicated in step 432. The imagesadvantageously have a significant overlap so that geometric information,e.g. geographic information, on one image can be transferred to asubsequently or previously captured image and so that relativeorientation changes can be detected. The overlap typically may beselected in the order of 60%, although embodiments of the presentinvention are not limited thereto. From the overlap of at least twoimages, tie points can be generated, as indicated in step 434. Such tiepoints are points occurring in the overlap of the images and thusallowing to determine a change in orientation of the instrument betweenacquisition of subsequent images. Furthermore, some ground controlpoints may be available, providing geographical information indicating ageographical correlation between objects in the region of interest andtheir image in the two dimensional image, e.g. via GPS, via a list ofpreviously recorded images, etc. The method may comprise a calibrationstep, wherein bundle adjustment is performed as indicated in 442, basedon the generated tie points, indicated in 438, on global positioningcoordinates, indicated in 440 and on initial camera parameters 436. Thispost processing step allows to obtain a more accurate exteriororientation, as indicated in 444, and which then can be used forobtaining corrected frame images having an accurate exteriororientation, as indicated in step 460. Optionally also accurate objectpoints and frame camera parameters can be used. Accurate object pointsand accurate calibration frame camera parameters as well as standardDigital Elevation Model (DEM) products can be obtained as indicated insteps 446, 448, 480.

On the other hand, using another sensor, spectral camera imageacquisition, e.g. hyper-spectral camera image acquisition is performedin step 410, resulting in a set of spectral images as indicated in step412, whereby, in the present example each spectral image consists of aplurality of lines and each line contains information of a particularspectral band. As set out with reference to FIG. 3, the full spectralinformation regarding a region of interest for a given wavelength or ina given wavelength region is distributed over different, typicallysubsequently imaged, hyper-spectral images and using spectral splittingas indicated by 414, spectral plane information is obtained for the fullregion of interest as indicated in steps 416 a, 416 b. Using thegeometric-referencing information obtained in step 460,geometric-referenced multi-spectral information can be obtained bycoupling the geometric-referencing information including e.g.orientational information, to the spectral plane data, optionallyincluding calibrated hyper-spectral camera parameters as indicated in462. The latter results in geometric-referenced spectral information, asshown in 418 a, 418 b.

The aforementioned interpolation step takes place prior to thegeometric-referencing 462, i.e. at the stage 444. The interpolation isschematically illustrated in the more detailed flow chart in FIG. 7.

Using the obtained data, an orthorectification of the images may beperformed as indicated in steps 420 and 450 for the multi-spectral andconventional 2-dimensional image respectively, resulting in anorthophoto for both the multi-spectral and conventional 2-dimensionalimage, as indicated in steps 422 and 452 respectively.Orthorectification means terrain corrected geometric referencing ofimagery using for example the sensor exterior orientation parameters,frame camera parameters (also referred to as interior orientation) andstandard Digital Elevation Model (DEM) products. The result of thisoperation is an orthophoto. Combining these orthophoto images allowsperforming PAN sharpening of the multi-spectral data, as indicated instep 470, such that a PAN sharpened hyperspectral orthophoto can beobtained, as indicated in step 472. The orthorectification of theconventional 2-dimensional image may give rise to a digital surfacemodel, as indicated in step 454.

The above schematic overview illustrates some standard and optionalfeatures and advantages according to embodiments of the presentinvention.

The inventors have further found that the performance of processingsteps that rely on information from different spectral images can beimproved by providing an optional preliminary renormalization step. Thispreliminary renormalization step may comprise dividing spectral imagesin identically arranged areas; for each of the areas, calculating apredetermined characteristic across said set of images; and, for each ofthe images, renormalizing intensity values in each of the areas infunction of the predetermined characteristic of said area. For the saidareas, one or more representative characteristics of the intensityvalues can be calculated. The average intensity value over the area isone such characteristic. Another useful characteristic is the standarddeviation of the intensity values, which gives an indication of thecontrast which will be measured. More generally, the distribution of theintensity values could be calculated and represented in a larger set ofcharacteristics. The set of obtained characteristics per area can beused as normalization coefficients. After applying normalization usingthe characteristics, the values of those characteristics become uniformover different areas in in the resulting images.

The procedure to determine the normalization coefficients is carried outby averaging over a sufficiently large set of images, in order toaverage out the effect of the image content. Afterwards, thenormalization can be carried out using the established coefficients,either on the same images, or on other images acquired in a similar waywith the same instrument. This procedure simplifies the way of workingas it is not necessary to calculate new coefficients for every new setof images.

The use of pre-processing is inter alia based on the insight of theinventors that there are two components to the difference in intensityof a given physical feature between different spectral images of thesame acquisition series, which represent the physical feature indifferent wavelength bands: (1) the physical feature may have adifferent reflectivity in different wavelength bands and (2) the sensormay have a different sensitivity in different wavelength bands. Thesecond factor can be compensated by renormalizing the various parts ofthe images relative to an average value that is representative for eachrespective part. While it is not possible to compensate for the firstfactor, the inventors have surprisingly found that the efficiency ofregistration algorithms and the like already greatly improves aftercompensating the second factor alone. The effect is believed to be dueto the fact that real-world physical objects typically exhibit a slowlyvarying reflectivity in function of wavelength over a large part of thespectrum of interest.

The predetermined characteristic may be an average intensity, and therenormalizing may comprise renormalizing the intensity values in each ofthe areas relative to the average intensity value.

The areas may correspond to individual pixels. It is an advantage ofthis embodiment that the sensor is effectively calibrated on a per-pixelbasis, such that variations in sensitivity of individual pixel-filtercombinations can be accounted for, regardless of the source of suchvariations (including manufacturing tolerances or impurities in thefilter). This leads to a maximal suppression of artefacts. By adding anoptical system to the pixel-filter combinations, a complete imagingsystem is obtained. It can be chosen to include sensitivity variationscaused by the optical system to correct for those, or to exclude them sothat the system remains generic for different optical systems.

Alternatively, the areas may correspond to distinct wavelength bands. Itis an advantage of this embodiment that the renormalization can beperformed per block of pixels, wherein a block typically represents arectangular strip of the sensor or a combination of multiple rectangularareas.

Where the examples of embodiments of the present invention mainly referto geometric referencing for aerial photography, aerial imaging orsatellite imaging, as indicated above, embodiments of the presentinvention are not limited thereto and may for example also be used forindustrial inspection etc. In one example a sensing device can forexample be used for inspecting goods on a conveyor belt, e.g. fordetecting foreign materials between goods or for detecting deviatinggoods. Such foreign materials or deviating goods typically will show aspectral image deviating from the expected spectral image. The geometricreferencing information may be a lateral position of objects ormaterials but also may be a height or relative height. Such a height orrelative height of objects may for example be determined from thegeometric referencing information based on the viewing angle of thegeometric referencing sensor with respect to the object imaged. Derivingheight information from image data based on a known sensor position andviewing angle with respect to the overall region of interest to beimaged is known by persons skilled in the art.

In one aspect, the present invention also relates to a processing systemwherein the method for sensing or imaging or part of such method asdescribed in embodiments of the previous aspects are implemented in asoftware based manner. Such a processing system may include at least oneprogrammable processor coupled to a memory subsystem that includes atleast one form of memory, e.g., RAM, ROM, and so forth. It is to benoted that the processor or processors may be a general purpose, or aspecial purpose processor, and may be for inclusion in a device, e.g., achip that has other components that perform other functions. Thus, oneor more aspects of embodiments of the present invention can beimplemented in digital electronic circuitry, or in computer hardware,firmware, software, or in combinations of them. The processing systemmay include a storage subsystem that has at least one disk drive and/orCD-ROM drive and/or DVD drive. In some implementations, a displaysystem, a keyboard, and a pointing device may be included as part of auser interface subsystem to provide for a user to manually inputinformation. Ports for inputting and outputting data also may beincluded. More elements such as network connections, interfaces tovarious devices, and so forth, may be included. The various elements ofthe processing system may be coupled in various ways, including via abus subsystem. The memory of the memory subsystem may at some time holdpart or all of a set of instructions that when executed on theprocessing system implement the steps of the method embodimentsdescribed herein.

The present invention also includes a computer program product whichprovides the functionality of any of the methods according to thepresent invention when executed on a computing device. Such computerprogram product can be tangibly embodied in a carrier medium carryingmachine-readable code for execution by a programmable processor. Thepresent invention thus relates to a carrier medium carrying a computerprogram product that, when executed on computing means, providesinstructions for executing any of the methods as described above. Theterm “carrier medium” refers to any medium that participates inproviding instructions to a processor for execution. Such a medium maytake many forms, including but not limited to, non-volatile media, andtransmission media. Non volatile media includes, for example, optical ormagnetic disks, such as a storage device which is part of mass storage.Common forms of computer readable media include, a CD-ROM, a DVD, aflexible disk or floppy disk, a tape, a memory chip or cartridge or anyother medium from which a computer can read. Various forms of computerreadable media may be involved in carrying one or more sequences of oneor more instructions to a processor for execution. The computer programproduct can also be transmitted via a carrier wave in a network, such asa LAN, a WAN or the Internet. Transmission media can take the form ofacoustic or light waves, such as those generated during radio wave andinfrared data communications. Transmission media include coaxial cables,copper wire and fiber optics, including the wires that comprise a buswithin a computer.

While the invention has been illustrated and described in detail in thedrawings and foregoing description, such illustration and descriptionare to be considered illustrative or exemplary and not restrictive. Theinvention is not limited to the disclosed embodiments. Other variationsto the disclosed embodiments can be understood and effected by thoseskilled in the art in practicing the claimed invention. The foregoingdescription details certain embodiments of the invention. It will beappreciated, however, that no matter how detailed the foregoing appearsin text, the invention may be practiced in many ways, and is thereforenot limited to the embodiments disclosed. It should be noted that theuse of particular terminology when describing certain features oraspects of the invention should not be taken to imply that theterminology is being re-defined herein to be restricted to include anyspecific characteristics of the features or aspects of the inventionwith which that terminology is associated.

A single processor or other unit may fulfill the functions of severalitems recited in the claims. The mere fact that certain measures arerecited in mutually different dependent claims does not indicate that acombination of these measures cannot be used to advantage. A computerprogram may be stored/distributed on a suitable medium, such as anoptical storage medium or a solid-state medium supplied together with oras part of other hardware, but may also be distributed in other forms,such as via the Internet or other wired or wireless telecommunicationsystems. Any reference signs in the claims should not be construed aslimiting the scope.

While the invention has been described hereinabove with reference tospecific embodiments, this was done to clarify and not to limit theinvention. The skilled person will appreciate that various modificationsand different combinations of disclosed features are possible withoutdeparting from the scope of the invention.

The invention claimed is:
 1. A sensing device for obtaining geometricreferenced multi-spectral image data of a region of interest in relativemovement with respect to the sensing device, the sensing devicecomprising: at least a first two dimensional sensor element, the sensingdevice being adapted for obtaining subsequent multi-spectral imagesduring said relative motion of the region of interest with respect tothe sensing device thus providing distinct spectral information fordifferent parts of a region of interest using the first sensor element;a second two dimensional sensor element, the sensing device beingadapted for providing, using the second sensor element, an image of theregion of interest for generating geometric referencing information tobe coupled to the distinct spectral information; the first twodimensional sensor element being operable to capture a first sequence offrames at a first frame rate and the second two dimensional sensorelement being operable to capture a second sequence of frames at asecond frame rate; wherein the first frame rate is higher than thesecond frame rate; and wherein the sensing device further comprises aprocessor configured to generate intermediate geometric referencinginformation to be coupled to frames of said first sequence of frames forwhich no synchronous frame from said second sequence of frames isavailable, said intermediate geometric referencing information beingderived from one or more temporally adjacent frames from said secondsequence of frames.
 2. The sensing device according to claim 1, whereinsaid second frame rate is selected to ensure a predetermined amount ofoverlap between respective regions imaged by consecutive frames of saidsecond sequence of frames.
 3. The sensing device according to claim 1,wherein a spectral filter and the first sensor element are arranged forobtaining spectral information at a first wavelength or wavelength rangeusing a part of the first sensor element and for obtaining spectralinformation at a second wavelength or wavelength range using anotherpart of the first sensor element.
 4. The sensing device according toclaim 1, wherein the first sensor element and second sensor element areintegrated on the same substrate.
 5. An imaging system comprising thesensing device according to claim
 1. 6. An aerial vehicle comprising theimaging system according to claim
 5. 7. A method for obtaining geometricreferenced multi-spectral image data of a region of interest in relativemovement with respect to a sensing device, the sensing devicecomprising: obtaining subsequent multi-spectral images during saidrelative motion of the region of interest with respect to the sensingdevice thus providing distinct spectral information for different partsof a region of interest using the first sensor element; providing, usingthe second sensor element, an image of the region of interest forgenerating geometric referencing information to be coupled to thedistinct spectral information; the first two dimensional sensor elementcapturing a first sequence of frames at a first frame rate and thesecond two dimensional sensor element capturing a second sequence offrames at a second frame rate; wherein the first frame rate is higherthan the second frame rate; and wherein the method further comprisesgenerating intermediate geometric referencing information to be coupledto frames of said first sequence of frames for which no synchronousframe from said second sequence of frames is available, saidintermediate geometric referencing information being derived from one ormore temporally adjacent frames from said second sequence of frames. 8.A computer program product comprising code means configured to cause aprocessor to carry out the method according to claim 7.